Evaluation of RDMA Over Ethernet Technology for Building Cost Effective Linux Clusters

Remote Direct Memory Access (RDMA) is an effective technology for reducing system load and improving performance. Recently, Ethernet offerings that exploit RDMA technology have become available that can potentially provide a high-performance fabric for MPI communications at lower cost than other competing technologies. The goal of this paper is to evaluate RDMA over gigabit Ethernet (ROE) as a potential Linux cluster interconnect. We present an overview of current RDMA technology from Ammasso, describe our performance measurements and experiences, and discuss the viability of using ROE in HPC applications. In a series of point-to-point tests, we find that the RDMA interface provides higher throughput and lower latency than legacy gigabit Ethernet. In addition, even when functioning in non-RDMA mode, the ROE cards demonstrate better performance than the motherboard network interfaces. For application benchmarks, including LINPACK and a climate model, the Ammasso cards provide a speedup over standard gigabit Ethernet even in small node configurations.